Learning Algorithms, Input Distributions and Generalization

We study the interaction between input distributions, learning algorithms and nite sample sizes in the case of learning classiication tasks. Focusing on the case of normal input distributions, we use statistical mechanics techniques to calculate the empirical and expected (or generalization) errors for several well-known algorithms learning the weights of a single-layer perceptron. In the case of spherically symmetric distributions within each class we nd that the simple Hebb algorithm is optimal. Moreover, we show that in the regime where the overlap between the classes is large, algorithms with low empirical error do worse in terms of generalization, a phenomenon known as over-training.